Description

When performing TSP/CP with packed sequences, the segmentation mask produced for attention from ids could introduce a large all-to-all operation (transporting bytes on the order of O(S^2)). This PR avoids that by introducing an all-gather on the small non-broadcasted to reduce communication.

More specifically, this line

mask = decoder_segment_ids[:, :, None] == decoder_key_segment_ids[:, None, :]

would produce an all-to-all on the broadcasted decoder_key_segment_ids[:, None, :] when running with TSP/CP since decoder_segment_ids is [B, S] following the input data sharding. The sequence dimension is sharded under TSP/CP constraints; however, this introduces a large all-to-all collective when constructing the mask.

To fix this, we add a sharding constraint on the right hand side of the equation. This all-gathers decoder_key_segment_ids on the sequence dimension for the right-hand side prior to broadcasting for a much smaller collective with the same sharded output. This reduces communication latency and can improve performance when training with large sequence lengths.

Tests

I observed all-to-all operations (300-400us) in the HLO for a Llama3-70b training run executing with tensor_sequence_parallelism. Then, after running the same configuration but with the new changes, I did not observe an all-to-all instruction in the HLO, and instead, the operation was replaced with a small all-gather (30us latency).

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.